In recent years, studies about mixed reality that aims at seamless joint of real and virtual spaces have been extensively made. An image display apparatus which presents mixed reality is implemented by superimposing an image of a virtual space (e.g., a virtual object, text information, and the like rendered by computer graphics) onto an image of a real space captured by an image sensing device such as a video camera or the like.
As applications of such image display apparatus, new fields different from conventional virtual reality such as operation assistance that superimposes the state in a body onto the body surface of a patient, a mixed reality game in which a player fights against virtual enemies that fly in the real space, and the like are expected.
A common requirement for these applications involves the accuracy of registration between the real and virtual spaces, and many efforts have been conventionally made in this respect.
A problem of registration in mixed reality amounts to obtaining the three-dimensional (3D) position and orientation of an image sensing device on a world coordinate system set on the real space (to be simply referred to as a world coordinate system hereinafter). As a method of solving these problems, it is a common practice to use a 3D position and orientation sensor such as a magnetic sensor, ultrasonic wave sensor, and the like.
In general, the output value of a 3D position and orientation sensor indicates the position and orientation of a measurement point on a sensor coordinate system which is uniquely defined by the sensor, but is not that of the image sensing device on the world coordinate system. Taking the Polhemus FASTRAK (magnetic sensor) which comprises a transmitter and receiver as an example, the position and orientation of the receiver on a coordinate system defined by the transmitter are obtained as the sensor output. Therefore, the sensor output value cannot be directly used as the position and orientation of the image sensing device on the world coordinate system, and must undergo some calibration processes. More specifically, coordinate transformation that transforms the position and orientation of a measurement point into those of the image sensing device, and coordinate transformation that transforms the position and orientation on the sensor coordinate system into those on the world coordinate system are required. In this specification, information used to transform the output value of the sensor for measuring the position and orientation of the image sensing device into the position and orientation of the image sensing device on the world coordinate system will be referred to as calibration information. Further, calibration information required for a coordinate transformation that transforms the position and orientation on the sensor coordinate system into those on the world coordinate system will be referred to as World Transform, and calibration information required for a coordinate transformation that transforms the position and orientation of a measurement point into those of the image sensing device will be referred to as Local Transform.
In order to attain accurate registration between the real and virtual spaces, accurate calibration information must be set by some means. A virtual image which is accurately registered to the real space can be displayed only when the accurate calibration information is given.
Note that the storing form of the calibration information is not particularly limited as long as the information can define the position and orientation of one coordinate system viewed from the other coordinate system. For example, a 4×4 viewing transformation matrix that represents transformation from one coordinate system to the other coordinate system may be used. The position and orientation may be expressed by a total of six parameters, i.e., three parameters that describe the position, and three parameters which express the orientation using an Euler angle. Also, the orientation may be expressed by four parameters, i.e., a three-valued vector that defines the rotation axis, and a rotation angle about that axis, or may be expressed by three parameters that express the rotation angle by the magnitude of the vector which defines the rotation axis.
Furthermore, the position and orientation may be expressed by parameters which represent their inverse transformations (e.g., the position and orientation of the world coordinate system on the sensor coordinate system).
In any of these cases, the position and orientation of an object on a 3D space have only six degrees of freedom (three degrees of freedom for the position, and three degrees of freedom for the orientation). Hence, unknown parameters required for calibration in this image display apparatus are a total of 12 parameters, i.e., six parameters required for transformation from the sensor coordinate system to the world coordinate system, and six parameters required for transformation from the position and orientation of the measurement point to those of the image sensing device.
As one of known methods for setting the calibration information, the user or operator interactively changes the 12 parameters (or 12 or more equivalent parameters) via some input means, and makes adjustment by trial and error until accurate registration is achieved.
Also, according to a calibration method proposed by Japanese Patent Laid-Open No. 2002-229730 (JP-A-2002-229730), if either the Local or World Transform values are obtained by some method, the remaining unknown parameters can be easily derived using a virtual image generated based on position and orientation information fixed to a given value as a visual queue.
However, the aforementioned conventional method requires complicated adjustment and long adjustment time.